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ABSTRACT 



Decision theory can be applied to four types ot decision situations in Education and 
Psyehology: selection, placement, classification, and mastery. For the application of the 
theory a utility function must be specified. Usually the utility function is chosen on a 
priori grounds. In this paper methods for the empirical assessment of utility functions 
by decision mikers are discussed. Moreover, methods for the analysis of the data are 
described. These methods are of two types: First, methods for the analysis of a 
subject's utility structure and, second, methods for investigating the type of utility 
functions. The procedures are demonstrated using an example from selection and an 
example from classification. It is concluded that decision makers can specify their own 
utility functions and that decision theory can be applied to practical situations in 
Education and Psyc v >logy. 
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Decision theory is a conceptual framework for making rational decisions. The 
theory was applied in Statistics (see, for example, Ferguson, 1967) and other 
sciences (French, 1986). The theory was introduced in psychometrics by Cronbach 
ari G!eser (1957), but was only further developed after the public, .ion of a paper of 
Hambieton and Novick (1973). 

In educational and psychological measurement four types of psychometric 
decision situations are distinguished (Van der Linden, 1985): (1) selection, (2) 
placement, (3) classification, and (4) mastery. In each of these situations the following 
elements are involved: First, one or more tests for gathering infonnation on subjects. 
Second, several treatments that differ among each other. Third, a decision rule to 
assign subjects to treatments. Fourth, a criterion to evaluate a subject's performance 
under a treatment. Finally, for the application of formal decision theory a utility function 
must be specified. 

In selection the subjects are usually assigned to two "treatments 11 : accept or 
reject for a job or educational program. The decision rule is of the following nature: 
accept applicants with testscores above one or more cutting scores on the test(s) and 
reject the others. At the end of a certain period the applicants are evaluated on a 
criterion, such as supervisors' ratings or grade point average. In placement the subjects 
are assigned to different treatments, where the performance is evaluated on the same 
criterion. An example is aptitude treatment interaction: Subjects with a low aptitude are 
assigned to another curriculum than subjects with a high aptitude; at the end of the 
curriculum all subjects are evaluated using the same achievement test as a criterion. In 
classification not only the treatments are different but aisc the subject's performance is 
evaluated on different criterions. An example in The Netherlands is the counseling for 
secondary education. Using achievement or intelligence tests students at the end of 
primary school are recommended different types of secondary education, varying from 
lower vocational to pre-univcrsity curriculums. These different types of secondary 
education are closed using completely different final examinations. The mastery 
situation has two "treaments": Pass or Fail. It differs from the other three situations in 
its criterion. In mastery the criterion is internal, i.e. th? true mastery of the subject 
matter, whereas the criterion of the other decision situations is external. 

The tools of formal decision theory are applied for making optimal decisions, 
such as the optimal assignment of subjects to treatments. As remarked before for 
applying formal decision theory a utility function must be specified. The utility 
function specifies the utility as a function of the criterion and the decision rule. In, for 
example, selection the utility is a function of the decision rule to accept or reject the 
applicant based on the testscore and the criterion score, such as success in the job or 
grade point average. 

The utility function can be specified ir three different ways. First, the function 
is specified on economic grounds. For example, in selection the utility of a rejected 
applicant has a constant negative value, i.e. the costs of testing an applicant. The utility 
of an accepted applicant represents the money value of the applicant, which is usually a 
nondecrtasing (constant or increasing) function of the applicant's criterion performance. 
Second, a mathematically convenient function is assumed Examples are the threshold 
(Hambieton & Novick, 1973), linear (Van der Linden & Mellenbergh, 1977), normal 
ogive (Novick & Lindley, 1978), the truncated normal and the extended beta (Chen & 
Novick, 1982) utility functions. Examples of the threshold, linear, and normal ogive 
utility functions are given in Figure 1. The third way of specifying utility functions is 



Insert Figure 1 about here 



empirical: The assessment of utility functions by decisionmakers themselves. The 
feasibility of this approach for mastery decisions in education was studied by Vrijhof, 
Mellenbergh, and Van den Brink (1983) and Van der Gaag, Mellenbergh, and Van den 
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Hrink (1988). Using this empirical approach u is possible to study whether the 
decisionmakers' utility functions resemble specific mathematical tonus such as the 
threshold or linear function. Vrijhof et al. (1983) and Van der Gaag et al. (1988) 
studied empirical utility functions for Pasr,/Fail decisions in educational testing They 
found that many of the empirical functions were adequately approximated by a linear 
function. J 

When the utility function is specified decision theory can be applied for making 
optimal decisions. The emphasis in the literature was on the optimal assignment of 
students and applicants to treatments. Examples are: Optimal cutting scores for 
Pass/Fail decisions (Hambleton & Novick, 1973; Huyhn, 1976; Mellenbergh 
KoppelaarA Van der Linden, 1977; Van der Linden & Mellenbergh, 1977- 
Veldhuyzen, 1982) and classification decisions (Van der Linden, 1987); optimal cutting 
scores for culture-fair selection (Gross & Su, 1975; Mellenbergh & Van der Linden, 
198 1; Petersen, 1976); optimal assignment of students to different treatments in 
Aptitude Treatment Interaction Research (Van der Linden, 1981). Moreover the 
construction of coefficients for tests (Van der Linden & Mellenbergh, 1978), a 
conceptual framework for the description and evaluation of the qualities of tests 
(Mellenbergh & Van der Linden, 1979), test length (Van der Linden, 1982), and item 
selection (Mellenbergh & Van der Linden, 1982) were studied from a decisiontheoretic 
point of view. 

In this paper recent results on the empirical assessment of utility functions are 
reported. First, the methods for assessing and analyzing these functions are described 
Subsequently, two examples are given. One on the assessment of utility in a selection 
situation and one in a classification situation. 



METHODS 

Several methods were used for the collection and analysis of data; they are 
described in this section. 



Empirical Assessment of Utility Functions 

The utility is a function of the decision rule based on the testscore and the 
criterion performance. For a given decision the utility is a function of only the criterion 
performance. For example, in selection the utility is a function of the decision rule to 
accept or reject the applicants and the applicant's criterion performance; for accepted 
subjects the utility is a function of criterion performance and for rejected applicants the 
utility is another function of criterion performance. When these functions are scaled 
separately for the different decisions a problem arises: the functions might be assessed 
in different scales. Therefore, also methods are needed to represent the different utility 
functions on identical scales. 

Three methods were used for the empirical assessment of utility functions- 
Comrey s constant sum method (Torgerson, 1958, p. 105), Bechtel's (1976) method, 
and a Direct Method (Lamers, Van der Gaag & Mellenbergh, 1988). 

For the application of Comrey's and Bechtel's methods a set of criterion 
performances must be prepared. For example, Lamers et al. (1988) studied a selection 
situation where the criterion performance was judged on thirteen aspects using five- 
pomt scales. The criterion score was the sum of the thirteen aspect scores, which ranged 
from 13 through 65. They used the following seven, equally spaced, criterion 
performance points: 19.5, 26, 32.5, 39, 45.5. 52, and 58.5. All possible pairs of the 
selected criterion performance points were formed The pairs were collected in a 
booklet; the order of the pairs was determined using Ross' (1934) table U«ing 
Bechtel's method the pairs were administered to the subjects for the acceptance 
situation. K 

Using Comrey's method the subject is asked to i ndicate for each member of a 
pair whether the utility is positive or negative. Then, the subject must split 100 points 
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according to the ratio of the utilities of the two members of the pairs. An example of the 
instruction is given in Figure 2> 



In^rt Fgure 2 through 5 about here 



The subject has indicated that the utility for both 39 and 52 is positive and (s)he has 
split the 100 points into 20 and 80, which means that the ratio of the two utilities is 
80/20 = 4. Using Bechtel's method a rating scale is used. An example is given in 
Figure 3. The subject has indicated to have a moderate preference for the criterion 
performance of 52. 

Using the Direct Method it is not needed to collect paired comparison judgments 
of the selected crl.erion performance points. Each of the points is rated; an example is 
given in Figure 4. The advantage is of course that much less judgements arc required. 
The number of ratings of seven criterion peformance points is seven, whereas the 
number of paired comparison is 21. The disadvantage of the Direct Method is that 
internal checks on the judgments are not possible, which will be discussed in the next 
section. 

As remarked before the utility functions of different decisions must be 
represented on the same scale. For the selection situation this problem was attacked by 
assessing the point of intersection of the two utility functions. The subjects were 
instructed to indicate the point of intersection of the utility functions of accepted and 
rejected applicants. The instruction is reported in Figure 5. 

Finally, it is remarked that in our most recent experiments a personal computer 
was used for collecting the data. One of the above mentioned methods was used for 
assessing the utility functions and the functions were graphically represented on the 
screen. The subjects were offered the opportunity to correct their functions, which 
means that both uncorrected and corrected functions can be analyzed. 



Data Analysis 

The analysis of the data has two different aspects: First, a model for the 
preference judgments is specified and the utility functions are estimated under this 
model. Second, the form of the utility functions is investigated; for example, it is 
studied whether the functions are of the threshold, linear, or normal ogive type. 

Bechtel (1976, chap. 2) proposed a model for the judgments of his method; see 

Figure 3. A linear model for the ith subject's (i = l t 2 N) preference strength for 

the jth criterion performance compared to the kth criterion performance (J, k = 1,2 
n; j * k) is: 

dijk = u,j - u lk + yj k + e ijk , (1) 
with constraints 



n 

Xu.j = 0 (i = 1.2 N) , 

H 

n 

5>jk = 0 0=1,2 n) , 

k=l 



ykj = -yjk 0.k= 1,2 n;j*k) , 



(2) 

(3) 
(4) 
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Vjj = 0 (j = 1.2 n) . (5) 

In this model u tJ and are subject ith utilities for the jth t respectively, kth criterion 
performance, and e,^ is the residual term. The interaction term >jk is called the 
unscalability; it indicates that the preference strength is not additive in the utilities of the 
two criterion performances. If the interaction term is negligible a more simple model can 
be used: 

d,jk = u tJ - u ik +e lJk (6) 

with constraint Formula 2. The parameters in the models are estimated using the least 
squares method. 

The models of Formula 1 and 6 can be compared in two ways. First, the null 
hypothesis that the unscalability parameter y is zero can be tested using the F-statistic. 
Using this test it is assumed that the residuals e are independendy normally distributed 
with common variance (Bechtel, 1976, sec. 2.5.2 and 2.6.2). This assumption might 
be violated for paired comparison data. Therefore, product moment correlations can be 
used as descriptive measures of fit. Using the least squares estimates the reproduced 
values under model Formula 6 are: 

d ij* = * ■ (7) 

The correlations R s between observed dyk and reproduced d*^, computed overall N 
subjects and all distinct pairs of criterion performances, is a measure of fit for model 
Formula 6. Although the model may be i adequate overall description for a group of 
subjects, it is possible that the model does not fit for particular subjects. The correlation 
R s i, computed between djfe and d*jj k per subject, describes the fit of model Formria 6 
for each of the subjects. The correlation R u between the observed djfc and the under 
model Formula 1 reproduced values over all subjects and all distinct pairs of criterion 
scores is a descriptive measure of fit for the model Formula 1 . The measure R u is 
computed using an ANOVA-like table (Bechtel, 1976, pp. 26 and 27). R u is equal to or 
J^gf rthan R s because Formula 1 contains more parameters than Formula 6. The more 
R u differes from R s the larger the influence of the unscalability parameter y. 

Using Comrcy's method (Figure 2) the ratios of the absolute values of the utilities 
are scaled instead of the difference of the utilities. To apply models Formula 1 and 6 the 
logarithmic transformation of the utility ratios is used (Vrijfhof et al., 1983). Using the 
Direct Method (Figure 4) the models cannot be applied because the data are not of the 
paired comparison type. The disadvantage of this quick and easy method is that the 
utility structure cannot be investigated and that the structure of model Formula 6 must be 
assumed. 

For Comrcy's and Bechtel's methods the utility structure can be investigated. When 
model Formula 6 is a reasonable description of the data it makes sense to investigate the 
form of the utility functions. Otherwise, the interaction term y (Formula 1) is needed 
which means that the utility judgment also depends on the particular pair of criterion 
performances that is considered. In that case it does not make much sense to investigate 
the form of the utility functions. 

Suppose model Formula 6 is a reasonable description of the data. The utility 
functions are estimated under this model and the form of the functions can be studied. 
First, the functions are graphically represented and the graphs are inspected. Second, 
the fit of some of the apriori functions is investigated. The fit of the threshold function 
is assessed using the point biserial correlation. For example, Lamers et al. (1988) used 

in their study seven criterion performances: 19.5, 26 58.5. The fit of the threshold 

function was investigated by computing six point biserial correlations: between the 
utilities and the criterion performance 1 9.5 scored as 0 and the other six criterion 
performances ; scored as 1 ; between the utilities and the criterion performances 19.5 ard 
26 scored as 0 and the other five criterion performances scores as 1 ; and so on. The 
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highest of these squared point biserial correlations was used as a measure of fit of a 
threshold function. The fit of a linear function was assessed using the sqrared product 
moment correlation between the utilities and the criterion performances. The fit of the 
normal ogive function is investigated by reseating the utilities into the open interval 
between zero and one and applying the logit transformation to the rescaled utilities. The 
squared product moment correlation between these logit transformed utilities and the 
criterion performance is a measure of fit of the normal ogive function (Van derGaag, 
1989). 



EXAMPLES 

In thi^ section some results of two studies arc briefly described. 



Selection 

Lamers et al. (1988) studied the selection situation; some information on this 
experiment was already given in previous sections. 

The situation is a multi-stage selection procedure. In the first stage psychological 
tests are used in selecting applicants for a training. At the end of the practical part of the 
training applicants are evaluated on thirteen aspects that are scored on a five-point scale. 
The criterion score is the sum of the thirteen aspect scores (minimum: 13; maximum: 
65). 

Seven levels of the criterion performance were used, equally spaced over the score 

range: 19.5 (1.5 on average per aspect), 26 (2.0 on average) 58.5 (4.5 on 

average). The 21 distinct pairs of these performance levels were formed. Bechtel's 
method (Figure 3) was used for assessing the utility function for accepted applicants, 
the Direct Method (Figure 4) was used for the rejected applicants, and the intersection 
was directly assessed (Figure 5). 

The subjects were persons actually involved in the selection process: psychologists 
doing the selection of applicants for the training and judges doing the evaluation of the 
trainees' criterion performance of the practical part of the training. Data of four 
psychologists and six judges were analyzed. 

For Acceptance the model of Formula 1 was fitted separately to the group of judges 
and the group of psychologists. The null hypothesis that the unscalability parameter is 
zero was tested using the F-statistic. The null hypothesis was rejected at the 1% 
significance level in each of the two groups. As remarked before the assumptions of the 
statistical test might be violated. Therefore, also the descriptive measures of fit were 
computed (Table 1). Comparing the fit of models Formula 1 (R u ) and Formula 6 (R s ) 



Insert Table 1 about here 



shows that the influence of the unscalability parameter is substantial in each of the two 
groups. Although the fit of model Formula 1 is better than the fit of model Formula 6 
Formula 6 is a reasonable approximation for the utility structure of the psychologist*. 
The correlation for the group of psychologists is .907 and the lowest of the individual 
correlations is .830. For the judges model Formula 6 is a good approximation for no. 3 
and 4, might be sufficient for no. 1 and 6, but is insufficient for no. 2 and 5. 

For Rejection the Direct Method was used. Therefore, it is imiX>ssible to study 
the stucture of an utility model. 

Using the least squares method the utilities were estimated and the utility 
functions were plotted. The fit of a linear utility function was assessed by using the 
squared product moment correlation and the fit of a threshold utility function was 



assessed by using the highest of the six squared point biseriai relations The sauared 
correlations are reported in Table 2. For Acceptance in nine of the ten cases 5? fflf" 1 



Insert Table 2 about here 



Zr 22 ,c "iS? fU U,an the J hreshoId function - For of the ten subjects the linear 
V55 ^"P 00 " of *c ^lity function: the lowest squared product moment 
£ri£?S S ^' .57 RejeCU ° n in Six 0f the ten cases the Malfunction has a better 
?V?S ^K h r d fu 1 CtlCn - For a " of *■ Pathologists and four of the judges(no. 
hnf for ^ i f fu ?^ tl0n 15 3,1 adw * uate approximatior. of their utility function, 
but for the judges no. 2 and 6 the linear function is not adequate. Inspection of the linear 

E™ on ; ? f thC UtiU , ty funCti0n Sh0wed ** 3,1 func *> ns ^ ££p££ tend to 

S^w^ fOT RejeCti0n tend 10 decrease with ^^on performance. 
Examples of utility functions are given in Figures 6 (Acceptance) and 7 (Rejection) For 
the P^^ologist the linear approximation is adequate, whereas the aDproximation for 



Insert Figures 6 and 7 about here 



., . ^ P° int of intersection of the two utility functions was directly assessed The 
criterion performances where the two utility functions intersect vary from 30 to 40. 



Classification 



The classification situation was studied by Van der Heiden (1988) He 
investigated the recommendations based on achievement or intelligence testscores at the 

SSSSSSSST 1 ^aT" COmm ° n types of ^cond^ry education-Lower 
vocational Training (LBO), Lower Administrative Training (MAVO) Hieher 
Administrative Training (HAVO), and Pre-University Training (VWO> 
Ednca.ion^fX p< r? orTnancc wa , s the Achieved Level after Six Years of Secondary 
rtSS ? 1 ? f j evels were used: LBO graduated within five years 
MA vkifSFvn*^ W, i5 m f °u Ur (LB04) > MA V0 within five years 

^^VmSES^ W,t 5 m f °u Ur E™ (MA V04) > "A V0 within six 

years (HAVOo), HAVO graduated within five years (HAV05), and VWO graduated 
within six years (VWO). The curriculum of LBO and MAVO y^SffiRo 
five yeansand of VWO six years. LB05, MAV05, and HAV06 implvTat a class k 
repeated. The points of these levels on a scale Achieveniem Uvel after^k Yearsof 
Secondary Education were known from another study (Cremers 1980) 
B e chtei?L 2 rJ, < S StinCt Pa T S ? * e ■ e T ?f erion Pcrio^ances were formed and 
HAVO JTvSurR 3PP - ° r C f • ° f thc four commendations (LBO, MAVO, 
HAVO, and VWO) the pairs were administered to three groups of subjects- nine 
primary school teachers, nine secondary school teachers and ten parents of students at 
the end of primary school. The data were collected using a i2nSSf552«?Tte 
^-SSSSf" ^ ° f ^ UdUty ^--^hcy got the P opporS?ity to 

asses J n FollVhT^ endati ° ns) X 28 (S " bjeCts) = 1 12 utilit y func tions were 
assessed. For each of the functions seven utility values were estimated The mean 
number of corrections per function was 2.77, which means that abouUO^nTof the 
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n?™^ 5 W ! rc co,Tec i ed -.Morcover, per function the correlation coefficient between 
uncorrected and corrected utility values was computed. 99 out of the 112 correlations 
were above .80. 

,h. u F u T CJ !u h • f 1* S 1011 ?! ? f ^J 60 * 311(3 each of *c four recommendations 
he null hypothesis that the unscalability parameter is zero was tested. In each of the 
twelve cases the null hypothesis was rejected at the 5 percent significance level. But for 
most or the 1 12 uuhtv functions model Formula 6 is a reasonable description of the 
unuty structure: 75 of these correlation coefficients were above 80. 

The corrected and uncorrected utility functions of the first subject are shown in 
Mgure 8. For the highest recommendation (VWO) the utility is an increasing function of 



Insert Figure 8 about here 



foe criterion Achievement Level after Six Years of Secondary Education (ALSYSE) 
H>r the other three recommendations the function increases until a maximum value and 
then decreases. The maximum corresponds with the recommendation: For the LBC 
rerommendanon the utility is maximal for LBO graduated within four years; for the 
r .k °J~ommendaDon the utility is maximal for MAVO graduated Within four years- 
for the HAVO recommendation the utility is maximal for HAVO graduated within five 



CONCLUSIONS 

V 1 ! ^ sdtS of f e 5 tarisdcal tests show that the additive structure of the utilities 
(hormula 6) does not fit the paired comparison data of our experiments But, the 
goodness of fit statistics usually indicate that the jdaitive stucture is a reasonable 
approximauon of the utility structure. 

In our studies (Lamers et al., 1988; Van der Gaag, 1988; Vrijhof et al., 1983) it 
? at ^ hncar f " ncDon 15 "^^y ™ adequa« description of the type of the 
uuhty function; there are however, subjects that specify different types of functions 
*i „ * m J2 y ' 1 ^ i srctnarkcd that insight in the structure of the utilities and the type of 
udhty function might be of practical importance. When the utility function is of a given 
type, e.g. threshold, linear, or normal ogive, then decision theory can be applied to 
educational and psychological testing. w 
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Table 1 

Measures of Fit (Correlation) Utility Models for Acceptance in Selection 



Subje^K 




R u 


Correlation 

^s 


Rsi 


Psychologists 




.950 


.907 






no. 1 






.968 




no. 2 






.830 




no. 3 






.909 




no. 4 






.926 


Judges 




.841 


.756 






no. 1 






.770 




no. 2 






.451 




no. 3 






.875 




no. 4 






.934 




no. 5 






.620 




no. 6 






.748 



13 



Table 2 

Squared Product Moment Correlations Linear Utility Function 
and Optimal Squared Point Bisenal Correlations Threshold Function 



Subjects 



Situation 



Acceptance 



Rejection 



Linear Threshold 



Linear Threshold 



Psychologist 



Judge 



no. 1 


.984 


.741 (3) 


.905 


.823 (2) 


no. 2 


.983 


.775 (3) 


.988 


.828 (3) 


no. 3 


.910 


.920 (3) 


.815 


.952 (3) 


no. 4 


.966 


.859 (3) 


.897 


.865 (2) 


no. 1 


.869 


.832 (2) 


.892 


.845 (4) 


no. 2 


.790 


.770 (3) 


.526 


.768 (2) 


no. 3 


.985 


.794 (3) 


.973 


.816 (4) 


no. 4 


.968 


.845 (4) 


.785 


.787 (1) 


no. 5 


.921 


.792 (2) 


.974 


.832 (2) 


no. 6 


.888 


.738 (3) 


.625 


.765 (4) 



Note. Between parentheses the optimal threshold, e.g. the number 2 indicates that the 
criterion scores 19.5 and 26 are scored 0 and the other criterion scores arc scored I. 
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Accepted 



Rejected 




Figure 1 . Example of a (a) threshold (b) linear, 
and (c) normal ogive utility function, 
selection situation. 
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Figure 2 

Rxampic oi'Comrey's Method, Selection, Acceptance 

The applicant has been accepted: The decision has for me at the criterion 
performance level 

39 52 
an + + utility 

The utility ratio is 20 80 

Figure 3 

Example of Bechtel's Method, Selection, Acceptance 
Indicate the degree in which you prefer one option above the other 
An applicant is accepted and has a criterion score 
39 987654321012®456789 52 



Figure 4 

Example of the Direct Method, Selection, Rejection 
Indicate the degree in which you prefer one option above the other 

I evaluate the decision to reject an applicant with a criterion 
score of 39 

negative 98765432 101 23456789 positive 



Figure 5 

Instruction for Determining Point of Intersection, Selection 

At which of the criterion scores you equally evaluate the 
decision to accept or reject? It can be each of the scores 
between 13 and 65. At a score of ... I am indifferent whether 
the applicant is accepted or rejected. 
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Figure 6. Utility function and linear approximation (a) judge no. 2, 
and (b) psychologist no. 1, acceptance situation. 
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Figure 7. Utility function and linear approximation (a) Judge no. 3, 
and (b) psychologist no. 3, rejection situation. 
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Figure 8 . Uncorrected and corrected utility 

functions first subject, classification 
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